Dynamically adjusting a data compute node group

ABSTRACT

Some embodiments provide a novel method for load balancing data messages that are sent by a source compute node (SCN) to one or more different groups of destination compute nodes (DCNs). In some embodiments, the method deploys a load balancer in the source compute node&#39;s egress datapath. This load balancer receives each data message sent from the source compute node, and determines whether the data message is addressed to one of the DCN groups for which the load balancer spreads the data traffic to balance the load across (e.g., data traffic directed to) the DCNs in the group. When the received data message is not addressed to one of the load balanced DCN groups, the load balancer forwards the received data message to its addressed destination. On the other hand, when the received data message is addressed to one of load balancer&#39;s DCN groups, the load balancer identifies a DCN in the addressed DCN group that should receive the data message, and directs the data message to the identified DCN. To direct the data message to the identified DCN, the load balancer in some embodiments changes the destination address (e.g., the destination IP address, destination port, destination MAC address, etc.) in the data message from the address of the identified DCN group to the address (e.g., the destination IP address) of the identified DCN.

BACKGROUND

Load balancers are commonly used in datacenters to spread the trafficload to a number of available computing resources that can handle aparticular type of traffic. FIGS. 1 and 2 illustrate two commondeployments of load balancers in datacenters today. In FIG. 1, the loadbalancers 100 are topologically deployed at the edge of the network andbetween different types of VMs (e.g., between webservers 105 andapplication servers 110, and between application servers 110 and thedatabase servers 115). The load balancers 100 are in some deploymentsstandalone machines (e.g., F5 machines) that perform load balancingfunctions. Also, in some deployments, the load balancers are servicevirtual machines (VMs) that are executing on the same host computingdevices that execute the different layers of servers that have theirtraffic balanced by the load balancers. FIG. 2 illustrates one suchdeployment of load balancers as service VMs (SVMs).

In the load balancer deployments of FIGS. 1 and 2, the load balancersserve as chokepoint locations in the network topology because theybecome network traffic bottlenecks as the traffic load increases. Also,these deployments require manual configuration of the load balancers andthe computing devices that send data packets to these load balancers inorder to allow the load balancers to properly receive and distribute theload balanced traffic. These deployments also do not seamlessly grow andshrink the number of the computing devices that receive the loadbalanced traffic, as the data traffic increases and decreases.

BRIEF SUMMARY

Some embodiments provide a novel method for load balancing data messagesthat are sent by a source compute node (SCN) to one or more differentgroups of destination compute nodes (DCNs). In some embodiments, themethod deploys a load balancer in the source compute node's egressdatapath. This load balancer receives each data message sent from thesource compute node, and determines whether the data message isaddressed to one of the DCN groups for which the load balancer spreadsthe data traffic to balance the load across (e.g., data traffic directedto) the DCNs in the group. When the received data message is notaddressed to one of the load balanced DCN groups, the load balancerforwards the received data message to its addressed destination. On theother hand, when the received data message is addressed to one of loadbalancer's DCN groups, the load balancer identifies a DCN in theaddressed DCN group that should receive the data message, and directsthe data message to the identified DCN. To direct the data message tothe identified DCN, the load balancer in some embodiments changes thedestination address (e.g., the destination IP address, destination port,destination MAC address, etc.) in the data message from the address ofthe identified DCN group to the address (e.g., the destination IPaddress) of the identified DCN.

By employing this inline load-balancing (LB) method, a source computenode does not have to be configured to address certain data messages toload balancers while foregoing such addressing for other data messages.This method can also seamlessly perform load balancing for severaldifferent DCN groups. In some embodiments, the source compute node andDCN group(s) are within one compute cluster in a datacenter.Accordingly, the method of some embodiments can seamlessly load balancedata messages that are sent to one or more DCN groups within a computecluster from source compute nodes in the compute cluster.

In some embodiments, the source compute node is a virtual machine (VM)that executes on a host, and the load balancer is another softwaremodule that executes on the same host. Other VMs also execute on thehost in some embodiments. Two or more of the VMs (e.g., all of the VMs)on the host use the same load balancer in some embodiments, while inother embodiments, each VM on the host has its own load balancer thatexecutes on the host.

The host also executes a software forwarding element (SFE) in someembodiments. The SFE communicatively couples the VMs of the host to eachother and to other devices (e.g., other VMs) outside of the host. Insome embodiments, the load balancers are inserted in the egress path ofthe VMs before the SFE. For instance, in some embodiments, each VM has avirtual network interface card (VNIC) that connects to a port of theSFE. In some of these embodiments, the load balancer for a VM is calledby the VM's VNIC or by the SFE port to which the VM's VNIC connects. Insome embodiments, the VMs execute on top of a hypervisor, which is asoftware layer that enables the virtualization of the shared hardwareresources of the host. In some of these embodiments, the hypervisorprovides the load balancers that provide the inline load balancingservice to its VMs.

The load balancing method of some embodiments is implemented in adatacenter that has several hosts executing several VMs and loadbalancers. In some of these embodiments, some or all of the loadbalanced DCNs are other VMs that are executing on the same or differenthosts as the SCN VMs. Examples of source and destination compute nodesthat can be load balanced by the load balancing method of someembodiments include data compute end nodes (i.e., source and datacompute end nodes) that generate or consume data messages, or middleboxservice nodes that perform some type of data processing on the datamessages as these messages are being relayed between the data computeend nodes. Examples of data compute end nodes (DCEN) include webservers,application servers, database servers, etc., while example of middleboxservice nodes include firewalls, intrusion detection systems, intrusionprevention systems, etc.

In a multi-host environment of some embodiments, the load balancers onthe host implement a distributed load balancing (DLB) method. This DLBmethod of some embodiments involves deploying one or more load balancerson the hosts that execute the SCN VMs. The load balancers on the hostsenforce the load balancing rules needed to spread the data traffic fromthe SCN VMs on their hosts to the DCNs of one or more DCN groups. Inthis distributed implementation, each load balancer enforces just theload balancing rules that are applicable to its SCN VM or VMs.

A set of one or more controllers facilitate the DLB operations of someembodiments. For instance, in some embodiments, the load balancers onthe hosts collect data traffic statistics based on the data messagesthat they load balance. These load balancers then pass the collectedstatistics to the controller set, which aggregates the statistics. Insome embodiments, the controller set then distributes the aggregatedstatistics to load balancing agents that execute on the hosts. Theseagents then analyze the aggregated statistics to generate and/or toadjust load balancing criteria that the load balancers (that execute onthe same hosts as the agents) enforce. In other embodiments, thecontroller set analyzes the aggregated statistics to generate and/or toadjust load balancing criteria, which the controller set thendistributes to the hosts for their load balancers to enforce. In stillother embodiments, the controller set generates and distributes someload balancing criteria based on the aggregated statistics, while alsodistributing some or all aggregated statistics to the hosts so thattheir LB agents can generate other load balancing criteria.

Irrespective of the implementation for generating the load balancingcriteria, the collection and aggregation of the data traffic statisticsallows the load balancing criteria to be dynamically adjusted. Forinstance, when the statistics show that one DCN is too congested withdata traffic, the load balancing criteria can be adjusted dynamically toreduce the load on this DCN while increasing the load on one or moreDCNs in the same DCN group. In some embodiments, the collection andaggregation of the data traffic statistics also allows the DLB method toreduce the load in any load balanced DCN group by dynamicallyinstantiating or allocating new DCN VMs for the DCN group or byinstantiating or allocating new SCN VMs.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description, the Drawings and the Claims isneeded. Moreover, the claimed subject matters are not to be limited bythe illustrative details in the Summary, Detailed Description and theDrawing.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purposes of explanation, several embodiments of theinvention are set forth in the following figures.

FIGS. 1 and 2 illustrate two common deployments of load balancers indatacenters today.

FIG. 3 illustrates a load balancing architecture that employs the inlineload-balancing method of some embodiments.

FIG. 4 illustrates an example of inline load balancers.

FIG. 5 illustrates an example of a controller set that gathersstatistics from hosts and based on the gathered statistics, dynamicallyadjusts the load balancing operations.

FIG. 6 illustrates a more detailed architecture of a host that executesthe load balancing modules of some embodiments of the invention.

FIGS. 7 and 8 presents examples of load balancing rules of someembodiments.

FIG. 9 illustrates a process that a load balancer performs in someembodiments.

FIGS. 10 and 11 illustrate two processes that a load balancing agentperforms in some embodiments.

FIG. 12 illustrates a process that a controller set performs in someembodiments.

FIG. 13 illustrates a process that shows the operation of the controllerset for embodiments in which the controller set analyzes the membershipupdates and/or global statistics, and in response to this analysisspecifies and/or updates LB rules if needed.

FIGS. 14-16 present several examples that illustrate how someembodiments dynamically adjust the spreading of traffic by adjusting theload balancing criteria and by adding/removing DCN VMs.

FIG. 17 illustrates that the distributed load balancing architecture ofsome embodiments can be used to load balance the data traffic to andfrom middleboxes.

FIG. 18 presents an example that illustrates one VM's inline loadbalancer forming multiple distributed load balancers with multiple otherinline load balancers of other VMs.

FIG. 19 presents an example to illustrate that the distributed loadbalancers of some embodiments can differently translate the virtualaddresses of data messages to different groups of DCNs.

FIG. 20 illustrates a set of distributed load balancers that directwebserver data messages to a group of application servers to either ahigh-priority sub-group of application servers or a low priority groupof application servers based on the assessed priority of the datamessages.

FIG. 21 conceptually illustrates a computer system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

Some embodiments provide a novel method for load balancing data messagesthat are sent by a source compute node (SCN) to one or more differentgroups of destination compute nodes (DCNs). In some embodiments, themethod deploys a load balancer in the source compute node's egressdatapath. This load balancer receives each data message sent from thesource compute node, and determines whether the data message isaddressed to one of the DCN groups for which the load balancer spreadsthe data traffic to balance the load across (e.g., data traffic directedto) the DCNs in the group. When the received data message is notaddressed to one of the load balanced DCN groups, the load balancerforwards the received data message to its addressed destination. On theother hand, when the received data message is addressed to one of loadbalancer's DCN groups, the load balancer identifies a DCN in theaddressed DCN group that should receive the data message, and directsthe data message to the identified DCN. To direct the data message tothe identified DCN, the load balancer in some embodiments changes thedestination address (e.g., the destination IP address) in the datamessage from the address of the identified DCN group to the address(e.g., the destination IP address, destination port, destination MACaddress, etc.) of the identified DCN.

Examples of source and destination compute nodes that can be loadbalanced by the method of some embodiments include data compute endnodes (i.e., source and data compute end nodes) that generate or consumedata messages, or middlebox service nodes that perform some dataprocessing on the data messages that are relayed between the datacompute end nodes. Examples of data compute end nodes (DCEN) includewebservers, application servers, database servers, etc., while exampleof middlebox service nodes include firewalls, intrusion detectionsystems, intrusion protection systems, etc. Also, as used in thisdocument, a data message refers to a collection of bits in a particularformat sent across a network. One of ordinary skill in the art willrecognize that the term data message may be used herein to refer tovarious formatted collections of bits that may be sent across a network,such as Ethernet frames, IP packets, TCP segments, UDP datagrams, etc.

By employing the inline load-balancing (LB) method of some embodiments,a source compute node does not have to be configured to address certaindata messages to load balancers while foregoing such addressing forother data messages. In some embodiments, the service gets deployed fora SCN automatically when the SCN is deployed as a virtual machine on ahost, and the VM deployment process configures the load balancingcriteria for the VM. This method can also seamlessly perform loadbalancing for several different DCN groups. In some embodiments, theSCNs and the DCNs are within one compute cluster in a datacenter.Accordingly, the method of some embodiments can seamlessly load balancedata messages that are sent to one or more DCN groups within a computecluster from other source compute nodes in the compute cluster.

FIG. 3 illustrates a load balancing architecture 300 that employs theinline load-balancing method of some embodiments. This architecture is adistributed load balancing (DLB) architecture that has a load balancer305 in the egress datapath of each of several compute nodes. The computenodes in this example fall into three groups of servers, which are webservers 310, application servers 315, and database servers 320. In someembodiments, the three groups of servers are three-tiers of servers thatare commonly found in a dataceneter.

As shown, a load balancer 305 is placed at the output of each web orapplication server in this example, so that webserver data traffic tothe application servers is load balanced, and the application serverdata traffic to the database servers is load balanced. Each loadbalancer enforces the load balancing rules needed to spread the datatraffic that is sent from the load balancer's corresponding sourcecompute node (e.g., source servers) to multiple destination computenodes (e.g., destination servers) that are part of one DCN group. Inother words, this distributed implementation allows each load balancerto enforce just the load balancing rules that are applicable to itssource compute node. Also, this distributed architecture does not haveany load balancer that is a chokepoint as it receives too much datamessages from one or more source compute nodes that prevent it fromtimely spreading the data messages from another source compute node.

In some embodiments, some or all of the source and destination computenodes are virtual machines (VMs) that executes on a host, and some orall of the load balancers are other software module that executes on thesame hosts as their source compute nodes. FIG. 4 illustrates an examplein which the load balancers 305 and the three groups of servers 310,315, and 320 of FIG. 3 are executing on six hosts 405-430 in adatacenter. In the example illustrated in FIG. 4, one LB executes oneach host for each web or application server that needs some of its datamessages load balanced. In other embodiments, however, one load balanceron a host load balances the output data messages of two or more of theVMs (e.g., all of the VMs) on the host. Even under this architecturethat uses one load balancer for two or more SCN VMs, the load balancersimplement a DLB scheme as each load balancer enforces just the loadbalancing rules that are applicable to SCN VM or VMs on that host.

FIG. 4 illustrates that in addition to the VMs and load balancers thatexecute on the hosts, each host also executes a software forwardingelement (SFE) 435 in some embodiments. The SFE 435 on a hostcommunicatively couples the VMs of the host to each other and to otherdevices outside of the host (e.g., VMs on other hosts) through one ormore other forwarding elements (e.g., one or more switches and routers)outside of the host. Examples of SFEs include software switches,software routers, etc.

As shown in FIG. 4, the load balancers in some embodiments are insertedin the egress path of the VMs before the SFE. For instance, in someembodiments, each VM has a virtual network interface card (VNIC) thatconnects to a port of the SFE. In some of these embodiments, the loadbalancer for a VM is called by the VNIC of the VM or by the SFE port towhich the VM's VNIC connects. In some embodiments, the VMs execute ontop of a hypervisor, which is a software layer that enables thevirtualization of the shared hardware resources of the host. In some ofthese embodiments, the hypervisors provide the load balancers thatprovide the inline load balancing service to its VMs.

FIG. 4 also shows each host to have two data storages 440 and 445. Thefirst data storage is an LB rule data storage 440 (e.g., database),while the second data storage is a STAT data storage 445. In someembodiments, the host's data storage 440 stores LB rules that specifythe IP addresses of the DCN VMs of the DCN groups that are load balancedby the host's load balancers. In some embodiments, the LB rule storages440 not only store the IP addresses of the DCN VMs but also stores theload balancing criteria (metrics) that the load balancers use to loadbalance the data traffic. While one LB rule storage 440 is shown for allload balancers 305 in FIG. 4, one of ordinary skill in the art willrealize that in other embodiments each load balancer 305 has its own LBrule storage 440.

In some embodiments, a SCN VM sends a data message to a virtual address(e.g., a virtual IP (VIP) address) that is associated with a loadbalanced DCN group. Before this data message is processed by the SFE ofthe VM's host, the SCN VM's load balancer intercepts the data messageand determines that it is addressed to a DCN group (e.g., determinesthat the message's destination IP address is the VIP of a DCN group)whose input data should be load balanced by the load balancer. The loadbalancer then replaces the virtual address in the data message with aDCN VM's physical address (e.g., the VM's IP address) that is stored inthe LB rule storage 440. The changing of the destination virtual addressto a DCN VM's physical address is a form of destination network addresstranslation. As the virtual address is replaced by a physical address,the virtual address does not have to be routed out of the host, whichsimplifies the deployment of the load balancing scheme.

In selecting the DCN VM that should receive the data message, the loadbalancer in some embodiments uses the load balancing criteria that isstored in the LB rule storage 440. After changing the network address ofthe received data message, the load balancer supplies the data messageto the SFE for it to process so that the data message can reach theaddressed DCN VM. One intrinsic advantage of this approach is that nosource address translation (e.g., source NAT) is required because thetraffic comes back to the SCN VM that generated the traffic.

The STAT data storage 445 stores statistics regarding the load balanceddata messages. For instance, as the load balancers 305 spread the datamessages to one or more load balanced DCN groups, the load balancers insome embodiments store statistics about how many data messages and/orhow many data flows are being sent to each DCN in each load balanced DCNgroup. In other embodiments, the load balancers store other statistics,as further described below. While one STAT data storage 445 is shown forall load balancers 305 in FIG. 4, one of ordinary skill in the art willrealize that in other embodiments each load balancer 305 has its ownSTAT data storage 445.

In some embodiments, the statistics that are stored in the STAT datastorage 445 on each host are passed to a set of one or more LBcontrollers that facilitate the DLB operations of some embodiments. Thecontroller set then aggregates the statistics that it receives from eachhost. The controller set then (1) distributes the aggregated statisticsto each host so that each host can define and/or adjust its loadbalancing criteria, and/or (2) analyzes the aggregated statistics tospecify and distribute some or all of the load balancing criteria forthe load balancers to enforce. In this manner, the load balancingcriteria can be dynamically adjusted based on the statistics that arestored in the STAT data storage 445.

In some embodiments, the controller set also dynamically instantiates orallocates VMs to SCN or DCN groups in order to reduce the load in anyload balanced DCN group. The controller set can also dynamicallyinstantiate or allocate VMs to SCN or DCN groups when it detects that aVM in one of these groups has crashed or has other operational issues.In such circumstances, the load balancing operations of the distributedload balancers can be adjusted in order to use the newly instantiated orallocated VM, and to reduce or eliminate the use of the VM that hascrashed or has operational issues.

FIG. 5 illustrates an example of a controller set that gathersstatistics from hosts and based on the gathered statistics, dynamicallyadjusts the load balancing operations. Specifically, this figureillustrates a multi-host system 500 of some embodiments. As shown, thissystem includes multiple virtualized hosts 505-515, a set of loadbalancing (LB) controllers 520, and a set of one or more VM managingcontrollers 525. As shown in FIG. 5, the hosts 505-515, the LBcontroller set 520, and the VM manager set 525 communicatively couplethrough a network 575, which can include a local area network (LAN), awide area network (WAN) or a network of networks (e.g., Internet).

The VM managing controllers 525 provide control and managementfunctionality for defining (e.g., allocating or instantiating) andmanaging one or more VMs on each host. These controllers in someembodiments also provide control and management functionality fordefining and managing multiple logical networks that are defined on thecommon software forwarding elements of the hosts. In some embodiments,the hosts 505-515 are similar to the hosts 405-430 of FIG. 4, exceptthat the hosts 505-515 each are shown to include an LB agent 560 forinteracting with the LB controller set 520, while not showing the othercomponents of the hosts, such as LB and STAT data storages 440 and 445.The LB agents 560 gather the collected statistics from the STAT datastorage 445, and relay these statistics to the LB controller set 520. Insome embodiments, the LB agents 560 aggregate and/or analyze some of thestatistics before relaying processed statistics to the LB controllerset, while in other embodiments the LB agents relay collected rawstatistics to the LB controller set.

The LB controller set 520 aggregates the statistics that it receivesfrom the LB agents of the hosts. In some embodiments, the LB controllerset 520 then distributes the aggregated statistics to the LB agents thatexecute on the hosts. These agents then analyze the aggregatedstatistics to generate and/or to adjust LB rules or criteria that theload balancers that execute on the same hosts as the agents enforce.

In other embodiments, the controller set analyzes the aggregatedstatistics to generate and/or to adjust LB rules or criteria, which thecontroller set then distributes to the hosts for their load balancers toenforce. In some of these embodiments, the controller set distributesthe same LB rules and/or criteria to each load balancer in a group ofassociated load balancers (i.e., in a group of load balancers thatdistribute the data messages amongst the DCNs of a group of DCNs), whilein other embodiments, the controller distributes different LB rulesand/or criteria to different load balancers in the group of associatedload balancers. Also, in some embodiments, the controller setdistributes updated LB rules and/or criteria to some of the loadbalancers in an associated group of load balancers, while notdistributing the updated LB rules and/or criteria to other loadbalancers in the associated group.

In still other embodiments, the controller set generates and distributessome load balancing rules or criteria based on the aggregatedstatistics, while also distributing some or all aggregated statistics tothe hosts so that their LB agents can generate other load balancingrules or criteria. One of ordinary skill in the art will realize thatthe LB rules and/or criteria are not always adjusted based on theaggregated statistics. Rather the LB rules and/or criteria are modifiedonly when the aggregated statistics require such modification.

Irrespective of the implementation for generating the LB rules, thecollection and aggregation of the data traffic statistics allows the LBrules or criteria to be dynamically adjusted. For instance, when thestatistics show one DCN as being too congested with data traffic, the LBrules or criteria can be adjusted dynamically for the load balancers ofthe SCNs that send data messages to this DCN's group, in order to reducethe load on this DCN while increasing the load on one or more other DCNsin the same DCN group. In some embodiments, the collection andaggregation of the data traffic statistics also allows the LB controllerset 520 to reduce the load on any DCN in a load balanced DCN group bydynamically directing the VM managing controller set 525 to instantiateor allocate new DCN VMs for the DCN group or by instantiating orallocating new SCN VMs.

FIG. 6 illustrates a more detailed architecture of a host 600 thatexecutes the load balancing modules of some embodiments of theinvention. As shown, the host 600 executes multiple VMs 605, an SFE 610,a set of one or more load balancers 615, an LB agent 620, and apublisher 622. The host also has LB rule storage 440 and the STAT datastorage 445, as well as group membership data storage 684, policy datastorage 682, aggregated (global) statistics data storage 686, andconnection state storage 690.

The SFE 610 executes on the host to communicatively couple the VMs ofthe host to each other and to other devices outside of the host (e.g.,other VMs on other hosts) through one or more forwarding elements (e.g.,switches and/or routers) that operate outside of the host. As shown, theSFE 610 includes a port 630 to connect to a physical network interfacecard (not shown) of the host, and a port 635 to connect to the VNIC 625of each VM. In some embodiments, the VNICs are software abstractions ofthe physical network interface card (PNIC) that are implemented by thevirtualization software (e.g., by a hypervisor). Each VNIC isresponsible for exchanging data messages between its VM and the SFE 610through its corresponding SFE port. As shown, a VM's egress datapath forits data messages includes (1) the VM's VNIC 625, (2) the SFE port 635that connects to this VNIC, (3) the SFE 610, and (4) the SFE port 630that connects to the host's PNIC.

Through its port 630 and a NIC driver (not shown), the SFE 610 connectsto the host's PNIC to send outgoing packets and to receive incomingpackets. The SFE 610 performs message-processing operations to forwardmessages that it receives on one of its ports to another one of itsports. For example, in some embodiments, the SFE tries to use headervalues in the VM data message to match the message to flow based rules,and upon finding a match, to perform the action specified by thematching rule (e.g., to hand the packet to one of its ports 630 or 635,which directs the packet to be supplied to a destination VM or to thePNIC). In some embodiments, the SFE extracts from a data message avirtual network identifier and a MAC address. The SFE in theseembodiments uses the extracted VNI to identify a logical port group, andthen uses the MAC address to identify a port within the port group. Insome embodiments, the SFE 610 is a software switch, while in otherembodiments it is a software router or a combined softwareswitch/router.

The SFE 610 in some embodiments implements one or more logicalforwarding elements (e.g., logical switches or logical routers) withSFEs executing on other hosts in a multi-host environment. A logicalforwarding element in some embodiments can span multiple hosts toconnect VMs that execute on different hosts but belong to one logicalnetwork. In other words, different logical forwarding elements can bedefined to specify different logical networks for different users, andeach logical forwarding element can be defined by multiple SFEs onmultiple hosts. Each logical forwarding element isolates the traffic ofthe VMs of one logical network from the VMs of another logical networkthat is serviced by another logical forwarding element. A logicalforwarding element can connect VMs executing on the same host and/ordifferent hosts.

The SFE ports 635 in some embodiments include one or more function callsto one or more modules that implement special input/output (I/O)operations on incoming and outgoing packets that are received at theports. One of these function calls for a port is to a load balancer inthe load balancer set 615. In some embodiments, the load balancerperforms the load balancing operations on outgoing data messages thatare addressed to DCN groups whose input traffic is being spread amongthe DCNs in the group in order to reduce the load on any one DCN. Forthe embodiments illustrated by FIG. 6, each port 635 has its own loadbalancer 615. In other embodiments, some or all of the ports 635 sharethe same load balancer 615 (e.g., all the ports share one load balancer,or all ports that are part of the same logical network share one loadbalancer).

Examples of other I/O operations that are implemented by the ports 635include firewall operations, encryption operations, messageencapsulation operations (e.g., encapsulation operations needed forsending messages along tunnels to implement overlay logical networkoperations), etc. By implementing a stack of such function calls, theports can implement a chain of I/O operations on incoming and/oroutgoing messages in some embodiments. Instead of calling the I/Ooperators (including the load balancer set 615) from the ports 635,other embodiments call these operators from the VM's VNIC or from theport 630 of the SFE.

The load balancers 615 perform their load balancing operations based onthe LB rules that are specified in the LB rule storage 440. For avirtual address (e.g., VIP) of a load balanced DCN group, the LB rulestorage 440 stores a load balancing rule that specifies two or morephysical addresses (e.g., IP addresses) of DCNs of the group to which adata message can be directed. In some embodiments, this load balancingrule also includes load balancing criteria for specifying how the loadbalancer should spread the traffic across the DCNs of the groupassociated with a virtual address.

One example of such load balancing criteria is illustrated in FIG. 7,which presents examples of load balancing rules that are stored in theLB rule storage 440. As shown, this data storage includes multiple LBrules 700, with each LB rule associated with one load balanced DCNgroup. In this example, each load balance rule includes (1) a set ofdata-message identifying tuples 705, (2) several IP addresses 710 ofseveral DCNs of the load balanced DCN group, and (3) a weight value 715for each IP address.

Each rule's tuple set 705 includes the VIP address (as the destinationIP address) of the rule's associated DCN group. In some embodiments, thetuple set 705 only includes the VIP address. In other embodiments, thetuple set also includes other data message identifiers, such as sourceIP address, source port, destination port, and protocol, which togetherwith the destination IP address form the five-tuple header values. Insome embodiments, a load balancer searches a LB data storage bycomparing one or more message identifier values (e.g., the destinationIP address, or one or more of the five-tuple header values) to the ruletuple sets 705 to identify a rule that has a tuple set that matches themessage identifier values.

Each LB rule's IP addresses 710 are the IP addresses of the DCNs thatare members of the DCN group that has the VIP address specified in therule's tuple set 705. In some embodiments, the addresses of the DCNs aresupplied as a part of the data initially supplied by the controller set(e.g., in order to configure the load balancer) or are supplied insubsequent updates to the DCN group information that is provided by thecontroller set.

The weight values 715 for the IP addresses of each LB rule provides thecriteria for a load balancer to spread the traffic to the DCNs that areidentified by the IP addresses. For instance, in some embodiments, theload balancers use a weighted round robin scheme to spread the trafficto the DCNs of the load balanced DCN group. As one example, assume thatthe DCN group has five DCNs and the weight values for the IP addressesof these DCNs are 1, 3, 1, 3, and 2. Based on these values, a loadbalancer would distribute data messages that are part of ten new flowsas follows: 1 to the first IP address, 3 to the second IP address, 1 tothe third IP address, 3 to the fourth IP address, and 2 to the fifth IPaddress.

As further described below, the weight values for an LB rule aregenerated and adjusted by the LB agent 620 and/or LB controller set insome embodiments based on the LB statistics that the load balancersstore in the STAT data storage 445. To gracefully switch betweendifferent load balancing criteria, the LB rules in some embodimentsspecify time periods for different load balancing criteria of a LB rulethat are valid for different periods of time.

FIG. 8 illustrates an example of load balancing rules 800 with such timeperiod parameters. These LB rules are stored in the LB rule storage 440in some embodiments. Each LB rule 800 has one message identifying tuple805, one or more IP address sets 810, and one or more weight value sets815. Each IP address set 810 has two or more IP addresses, and eachweight value set 815 is associated with an IP address set and has oneweight value for each IP address in its associated IP address set.

In the example illustrated in FIG. 8, each rule has multiple sets of IPaddresses and multiple sets of weight values. Each set of IP addressesand its associated set of weight values represents one set of loadbalancing criteria. For each of these sets of load balancing criteria,each rule has a time value 820 that specifies the time period duringwhich the IP address set 810 and its associated weight value set 815 arevalid. For instance, in a LB rule, the time value for one IP address setmight specify “before 1 pm on Sep. 1, 2014,” while the time value foranother IP address set might specify “after 12:59 pm on Sep. 1, 2014.”These two time periods allow the load balancers to seamlessly switchfrom using one IP address set and its associated weight value set toanother IP address set and its associated weight value set at 1 pm onSep. 1, 2014. These two IP address sets might be identical and theymight only differ in their associated weight value sets. Alternatively,the two IP address sets might be different. Two IP address sets mightdiffer but have overlapping IP addresses (e.g., one set might have fiveIP addresses, while another set might have four of these five IPaddresses when one DCN is added or removed from a DCN group).Alternatively, two IP address sets might differ by having no IPaddresses in common.

In FIG. 8, the time period values and the weight values are used in theLB rules. One of ordinary skill in the art will realize that in otherembodiments, the LB rules do include the weight values, but include thetime values to allow the load balancer to gracefully switch betweendifferent sets of load balanced DCNs. As before, two DCN sets may differby having mutually exclusive DCNs, or they may differ by having one ormore DCNs in common and one or more DCNs not in common.

As shown in FIG. 6, the host also includes a connection state storage690 in which the load balancer stores data records that allow the loadbalancer to maintain connection state for data messages that are part ofthe same flow, and thereby to distribute data messages that are part ofthe same flow statefully to the same DCN. More specifically, whenever aload balancer identifies a DCN for a data message based on the message'sgroup destination address (e.g., the destination VIP), the load balancernot only replaces the group destination address with the DCN's address(e.g., with the DCN IP address), but also stores a record in theconnection state storage 690 to identify the DCN for subsequent datamessages that are part of the same flow. This record stores thedestination IP address of the identified DCN along with the datamessage's header values (e.g., the five tuple values). In someembodiments, for fast access, the connection data storage 690 is hashindexed based on the hash of the data message header values.

To identify a DCN for a received data message, the load balancer firstchecks the connection state storage 690 to determine whether it haspreviously identified a DCN for receiving data messages that are in thesame flow as the received message. If so, the load balancer uses the DCNthat is identified in the connection state storage. Only when the loadbalancer does not find a connection record in the connection statestorage 690, the load balancer in some embodiments examines the LB rulesin the LB rule storage 440 in order to identify a DCN to receive thedata message.

By searching the connection state storage 690 with the messageidentifiers of subsequent data messages that are part of the same flow,the load balancer can identify the DCN that it previously identified fora data message of the same flow, in order to use the same DCN for themessages that are part of the same flow (i.e., in order to statefullyperform its load balancing operation). In some embodiments, the loadbalancer also uses the connection state storage 690 records to replacethe DCN's destination address with the virtual group address (e.g., thegroup VIP address) on the reverse flow path when the load balancerreceives (from the SFE port 630 or 635) data messages sent by the DCN tothe SCN. After translating of the destination addresses of a datamessage in the reverse flow, the load balancer returns the data messageto the SFE port that called it, so that the SFE port can direct the datamessage to SCN VM.

In some embodiments, the connection state storage 690 is addresseddifferently than the LB data storage 440. For instance, as mentionedabove, the connection state storage 690 in some embodiments stores itsconnection-state records based on hashed message identifier values(e.g., five tuple identifier values), while not using such a hashaddressing scheme for the LB rule data storage 440. In some embodiments,the hashed values specify memory locations in the connection statestorage 690 that store the corresponding message-identifier sets.Because of this addressing scheme, the load balancer generates a hash ofthe message-identifier set to identify one or more locations in theconnection state storage 690 to examine for a matchingmessage-identifier set. In other embodiments, the LB rule data storage440 is also hash indexed based on the hash of the tuple set 705.

In FIG. 6, only one LB rule data storage 440 and only one connectionstate storage 690 are illustrated for all the load balancers 615. Inother embodiments, each load balancer has its own rule data storage 440and connection state storage 690. In yet other embodiments, the host hasseveral rule data storages 440 and connection state storages 690, buttwo or more load balancers can share a rule data storage or connectionstate storage (e.g., two load balancers that are balancing the load fortwo VMs that are part of the same logical network). As further describedbelow by reference to FIG. 18, each load balancer 615 having its ownrule data storage 440 and connection state storage 690 allows thesestorages to be smaller and easier to search more quickly.

In some embodiments, each time a load balancer 615 performs a loadbalancing operation on a data message (i.e., replaces the destinationvirtual address of the message to a destination address of a DCN), theload balancer updates the statistics that it maintains in the STAT datastorage 445 for the data traffic that it relays to the DCN that wasaddressed as part of its load balancing operation. Several examples ofstatistics were provided above and will be further described below.

In some embodiments, the LB agent 620 gathers (e.g., periodicallycollects) the statistics that the load balancers store in the STAT datastorage(s) 445, and relays these statistics to the LB controller set520. Based on statistics that the LB controller set 520 gathers fromvarious LB agents of various hosts, the LB controller set (1)distributes the aggregated statistics to each host's LB agent so thateach LB agent can define and/or adjust its load balancing criteria,and/or (2) analyzes the aggregated statistics to specify and distributesome or all of the load balancing criteria for the load balancers toenforce.

In some embodiments where the LB agent receives new load balancingcriteria from the LB controller set, the LB agent stores these criteriain the host-level LB rule storage 688 for propagation to the LB rulestorage(s) 440. In the embodiment where the LB agent receives aggregatedstatistics from the LB controller set, the LB agent stores theaggregated statistics in the global statistics data storage 686. In someembodiments, the LB agent 620 analyzes the aggregated statistics in thisstorage 686 to define and/or adjust the load balancing criteria (e.g.,weight values), which it then stores in the LB rule storage 688 forpropagation to the LB rule storage(s) 440. The publisher 622 retrieveseach LB rule that the LB agent 620 stores in the LB rule storage 688,and stores the retrieved rule in the LB rule storage 440 of the loadbalancer 615 that needs to enforce this rule.

The LB agent 620 not only propagates LB rule updates based on newlyreceived aggregated statistics, but it also propagates LB rules orupdates LB rules based on updates to DCN groups that it receives fromthe LB controller set 520. The LB agent 620 stores each DCN group'smembers that it receives from the LB controller set 520 in the groupdata storage 684. When a DCN is added or removed from a DCN group, theLB agent 620 stores this update in the group storage 684, and thenformulates updates to the LB rules to add or remove the destinationaddress of this DCN to or from the LB rules that should include oralready include this address. Again, the LB agent 620 stores suchupdated rules in the rule data storage 688, from where the publisherpropagates them to the LB rule storage(s) 440 of the load balancers thatneed to enforce these rules.

When a DCN is added to a DCN group, the updated LB rules cause the loadbalancers to direct some of the DCN-group data messages to the addedDCN. Alternatively, when a DCN is removed from a DCN group, the updatedLB rules cause the load balancers to re-direct data messages that wouldgo to the removed DCN, to other DCNs in the group. However, even after aDCN is intentionally designated for removal from a DCN group, a loadbalancer in some embodiments may continue to send data messages (e.g.,for a short duration of time after the removal of the DCN) to the DCNthat are part of prior flows that were directed to the DCN. This allowsthe DCN to be removed gradually and gracefully from the DCN group as theflows that it handles terminate. Some embodiments also achieve agraceful transition away from a DCN that should be removed from the DCNgroup by using time values to specify when different LB criteria for thesame LB rule should be used. Some embodiments also use such time valuesto gracefully add a new DCN to a DCN group.

In some embodiments, the LB agent 620 stores in the policy storage 682,LB policies that direct the operation of the LB agent in response tonewly provisioned DCN VMs and their associated load balancers, and/or inresponse to updated global statistics and/or adjusted DCN groupmembership. The policies in the policy storage 682 in some embodimentsare supplied by the LB controller set 520.

FIG. 9 illustrates a process 900 that the load balancer 615 performs insome embodiments. As shown, the process 900 starts when the loadbalancer receives (at 905) a data message from its corresponding SFEport 635. This port relays this message when it receives the datamessage from its VM. In some embodiments, the port relays the datamessage by passing to the load balancer a reference (e.g., a handle thatidentifies a location in memory that stores the data message) to thedata message.

Next, the process determines (at 910) whether the received datamessage's destination address is a virtual address (e.g., the VIPaddress) of a DCN group that the load balancer has to balance its input.To make this determination, the process 900 checks a table in the LBrule data storage 440 that stores the virtual addresses of the DCNgroups that the process load balances.

When the process determines (at 910) that the data message is notdirected to a load balanced virtual address, the process sends (at 915)the message along the message's datapath without performing anydestination address translation on the message. This operation (at 915)entails informing the SFE port 635 that called it, that the process hascompleted processing the VM data message. The SFE port 635 can thenhandoff the VM data message to the SFE 610 or can call another I/O chainoperator to perform another operation on the VM data message. After 915,the process ends.

On the other hand, when the process determines (at 910) that the datamessage is directed to a load balanced virtual address, the processdetermines (at 920) whether the connection state cache 690 stores arecord that identifies the DCN to which the data message should berouted. As mentioned above, each time a load balancer uses a LB rule todirect a new data message flow a DCN of a DCN group, the load balancerin some embodiments creates a record in the connection state cache 690to store the physical IP address of the DCN, so that when the loadbalancer receives another data message within the same flow (i.e., withthe same message-attribute set), it can route it to the same DCN that itused for previous data message in the same flow.

Also, as mentioned above, the connection-state cache 690 in someembodiments stores each flow's record based on hashed address valuesthat are hashed versions of the flow identifying attributes of the datamessage header values. This addressing scheme allows the load balancerto quickly search the cache 690. Hence, before searching the rule datastore 440, the load balancer first generates a hash value from themessage-attribute set of the received data message (e.g., a hash of themessage's five tuples) to identify one or more memory locations in thecache 690, and then uses this hash value to examine the memorylocation(s) to determine whether the cache stores a connection-flowrecord with a matching set of attributes as the received VM datamessage.

When the process 900 identifies (at 920) a record for the received datamessage's flow in the cache 690, the process (at 925) then replaces themessage's destination address (i.e., the virtual group address, such asthe VIP address) with the DCN destination address (e.g., with the DCN IPaddress) that is stored in the record in the cache 690. At 925, theprocess sends the address-translated data message along its datapath. Insome embodiments, this operation entails returning a communication tothe SFE port 635 (that called the load balancer to initiate the process900) to let the port know that the load balancer is done with itsprocessing of the VM data message. The SFE port 635 can then handoff thedata message to the SFE 610 or can call another I/O chain operator toperform another operation on the data message. At 925, the process 900also updates in some embodiments the statistics that it maintains inSTAT storage 445 for the DCN to which the message was addressed by theprocess 900. This update reflects the transmission of a new data messageto this DCN. After 925, the process 900 ends.

When the process 900 determines (at 920) that the connection cache 690does not store a record for the received data message's flow, theprocess 900 searches (at 930) the LB rule data store 440 to identify anLB rule for the data message received at 905. To identify the LB rule inthe data store 440, the process in some embodiments compares a set ofattributes of the received data message with the data-messageidentifying tuples (e.g., tuples 705 of FIG. 7) of the rules to identifya rule that has a tuple set that matches the message's attribute set. Insome embodiments, the process uses different message-attribute sets toperform this comparison operation. For instance, in some embodiments,the message attribute set includes just the destination IP address ofthe message (e.g., the VIP of the addressed DCN group), which was usedat 910 to determine whether the message is directed to a load balancedDCN group. In other embodiments, the message attribute set includesother attributes, such as one or more of the other five-tupleidentifiers (e.g., one or more of the source IP, source port,destination port, and protocol). In some embodiments, the messageattribute set includes logical network identifiers such as virtualnetwork identifier (VNI), virtual distributed router identifier (VDRI),a logical MAC address, a logical IP address, etc.

As mentioned above, each LB rule in some embodiments includes two ormore destination addresses (e.g., IP addresses 710), which are thedestination addresses (e.g., IP addresses) of the DCNs that are membersof the DCN group that has the virtual address (e.g., VIP address)specified in the rule's tuple set 705. When the process identifies an LBrule (at 930), it selects one of the destination addresses (e.g., IPaddresses) of the rule to replace the virtual address (e.g., the VIPaddress) in the message. Also, as mentioned above, each LB rule storescriteria for facilitating the process' selection of one of thedestination addresses of the LB rule to replace the message's virtualdestination identifier. In some embodiments, the stored criteria are theweight and/or times values that were described above by reference toFIGS. 7 and 8. Accordingly, in some embodiments, the process 900 selectsone of the matching rule's destination addresses based on the selectioncriteria stored in the rule.

After changing the destination address of the data message, the process(at 935) sends the data message along its datapath. Again, in someembodiments, this operation entails returning a communication to the SFEport 635 (that called the load balancer to initiate the process 900) tolet the port know that the load balancer is done with its processing ofthe data message. The SFE port 635 can then handoff the VM data messageto the SFE 610 or can call another I/O chain operator to perform anotheroperation on the VM data message.

After 935, the process transitions to 940, where in the connection cachedata store 690, it creates a record to identify the DCN (i.e., toidentify the DCN destination identifier) to use to forward data messagesthat are part of the same flow as the data message received at 905. Insome embodiments, this record is addressed in the cache 690 based on ahash value of the message-attribute set identified at 905. At 940, theprocess 900 also updates the statistics that it maintains in STATstorage 445 for the DCN to which the message was addressed by theprocess 900. This update reflects the transmission of a new data messageto this DCN. After 940, the process ends.

FIGS. 10 and 11 illustrate two processes that the LB agent 620 performsin some embodiments. FIG. 10 illustrates a process 1000 that the LBagent 620 performs each time that it receives updated group membershipsand/or global statistics from the LB controller set 520. As shown, theprocess 1000 starts (at 1005) when it receives from the LB controllerset 520 updated statistics for at least one DCN group and/or updatedmembership to at least one DCN group.

Next, the process 1000 determines (at 1010) whether the received updateincludes an update to the membership of at least one DCN group for whichthe LB agents generates and/or maintains the LB rules. If not, theprocess transitions to 1020. Otherwise, the process creates and/orupdates (at 1015) one or more records in the group membership storage684 to store the updated group membership that the process received at1005. From 1015, the process transitions to 1020.

At 1020, the process 1000 determines whether the received updateincludes updated statistics for at least one DCN group for which the LBagents generates and/or maintains the LB rules. If not, the processtransitions to 1030. Otherwise, the process creates and/or updates (at1025) one or more records in the global statistics storage 686 to storethe updated global statistics that the process received at 1005. From1025, the process transitions to 1030.

At 1030, the process initiates a process to analyze the updated recordsin the group membership storage 684 and/or the global statistics storage686 to update the group memberships (e.g., the IP addresses) and/or theload balancing criteria (e.g., the weight or time values) of one or moreLB rules in the host-level LB rule data storage 688. This analyzingprocess will be further described below by reference to FIG. 11. Fromthe host-level LB rule data storage 688, the publisher 622 propagateseach new or updated LB rule to the LB rule data storage(s) 640 of theindividual load balancer(s) 615 (on the same host) that needs to processthe new or updated LB rule. In publishing each new or updated LB rule,the publisher 622 does not publish the LB rule to the rule data storage640 of a load balancer (on the same host) that does not need to processthe rule.

After 1030, the process 1000 ends.

FIG. 11 illustrates a process 1100 that the LB agent 620 performs insome embodiments to analyze updated records in the group membershipstorage 684 and/or the global statistics storage 686, in order to updatethe group memberships (e.g., the IP addresses) and/or the load balancingcriteria (e.g., the weight or time values) of one or more LB rules inthe host-level LB rule data storage 688. In some embodiments, the LBagent performs an identical or similar process when the LB agent powersup (e.g., when its host powers up) to configure the LB rules of the loadbalancers on the host, and when a new SCN VM is instantiated on the hostto configure the LB rules of the instantiated VM's load balancer.

As shown, this process 1100 initially selects (at 1105) a load balancer615 on the LB agent's host. In some embodiments, the process selects (at1105) only load balancers that are affected by one or more of theupdated records that resulted in the performance of this process. Next,at 1110, the process selects a virtual address (e.g., a VIP) of a DCNgroup that the selected load balancer has to load balance. The processthen retrieves (at 1115) the stored statistics and group membership datafor the DCN group identified by the selected virtual address.

At 1120, the process analyzes the membership and statistic recordsretrieved at 1115. Based on this analysis, the process determineswhether the group memberships (e.g., the IP addresses) and/or the loadbalancing criteria (e.g., the weight or time values) of one or more LBrules in the host-level LB rule data storage 688 should be specifiedand/or modified for the selected load balancer. To perform thisanalysis, the process 1100 uses one or more policies that are specifiedin the policy storage 682. If the process determines that it shouldspecify or update the group's membership and/or the load balancingcriteria for the selected group, the process performs (at 1120) thisspecifying or updating, and then stores (at 1125) the specified orupdated the group's membership and/or load balancing criteria in one ormore LB rules that are stored in the LB data storage 688. As mentionedabove, the specified or updated LB rules in the host LB rule storage 688are distributed by the publisher 622 to the LB data storage 440 of anyload balancer that on the same host performs load balancing operationson the input traffic to the selected group. Several examples of updatingload balancing criteria and/or group membership will be described below.

After 1125, the process determines (at 1130) whether it has examined allvirtual group identifiers (i.e., all the DCN groups) that the selectedload balancer has to load balance. If not, it selects (at 1135) anothervirtual group identifier (i.e., another DCN group) and returns to 1115to perform operations 1115-1130 for this newly selected virtual groupidentifier. Otherwise, the process transitions to 1140, where itdetermines whether it has examined the updates for all the loadbalancers (e.g., whether it has examined all the load balancers affectedby the new or updated group membership and statistic data) on its host.If so, the process ends. If not, the process selects (at 1145) anotherload balancer on the same host as the LB agent, and then repeatsoperations 1110-1140 for this newly selected load balancer.

FIG. 12 illustrates a process 1200 that one or more LB controllers inthe LB controller set 520 perform in some embodiments. As shown, theprocess 1200 starts (at 1205) when it receives statistics from one ormore LB agents and/or receives membership updates for one or more DCNgroups. The process 1200 in some embodiments receives the groupmembership updates from another process of the LB controller set. Forinstance, the LB controller set informs the process 1200 that a new DCNVM has been added to or removed from a DCN group when it is informed bythe virtualization manager set 525 that a new VM has been created for orterminated from the DCN group.

After 1205, the process updates (at 1210) (1) the global statistics thatthe LB controller set 520 maintains based on the statistics received at1205, and/or (2) the group membership(s) that the LB controller set 520maintains based on the group updates received at 1205. Next, at 1215,the process determines based on the updated statistics whether it shouldhave one or more SCN or DCN VM specified for or removed from the group.For instance, when the updated statistics causes the aggregatedstatistics for a DCN group to exceed an overall threshold load value forthe DCN group, the process 1200 determines that one or more new DCNshave to be specified (e.g., allotted or instantiated) for the DCN groupto reduce the load on DCNs previously specified for the group.Similarly, when the updated statistics causes the aggregated statisticsfor one or more DCN in the DCN group to exceed a threshold load value,the process 1200 may determine that one or more new DCNs have to bespecified (e.g., allotted or instantiated) for the DCN group to reducethe load on the congested DCNs. Conversely, when the updated statisticsshows that a DCN in a DCN group is being underutilized or is no longerbeing used to handle any flows, the process 1200 determines (at 1215)that the DCN has to be removed for the DCN group.

When the process 1200 determines (at 1215) that it should have one ormore SCN or DCN VM added or removed for the group, the process requests(at 1220) the VM managing set 525 to add or remove this VM, and thentransitions to 1225. The process also transitions to 1225 when itdetermines (at 1215) that no SCN or DCN VM needs to be added or removedfor the group. At 1225, the process determines whether the time hasreached for it to distribute membership update and/or global statisticsthat the LB controller set maintains to one or more LB agents executingon one or more hosts.

In some embodiments, the process 1200 distributes membership updatesand/or global statistics on a periodic basis. In other embodiments,however, the process 1200 distributes membership update and/or globalstatistics for one or more DCN groups whenever this data is modified. Inaddition to requesting the addition or removal of a VM from a group, thegroup membership can change when a VM that is part of a group fails.Such VM failures would have to be relayed to the LB agents so that theycan modify the LB rules of their associated load balancers. In someembodiments, the membership update data that the process 1200distributes, differentiates a failed DCN from an intentionally removedDCN (i.e., a DCN that has not failed but has been removed from the DCNgroup). This differentiation allows a load balancer's operation to bedifferently modified for the failed DCN and the intentionally removedDCN. For the failed DCN, the load balancer stops using the failed DCN,while for an intentionally removed DCN, the load balancer in someembodiments can continue to use the removed DCN for a duration of timeafter receiving the membership update (e.g., for new flows up to aparticular time, or for previously received flows that are beingprocessed by the DCN). To cause the load balancer to stop using thefailed DCN, the connection records that specify the failed DCN in theload balancer's connection storage 690 are removed in some embodiments.

When the process determines (at 1225) that it does not need todistribute new data, it transitions to 1230 to determine whether it hasreceived any more statistic and/or membership updates for which it needsto update its records. If so, the process transitions back to 1210 toprocess the newly received statistic and/or membership updates. If not,the process transitions back to 1225 to determine again whether itshould distribute new data to one or more LB agents.

When the process determines (at 1225) that should distribute membershipupdate(s) and/or global statistics, it distributes (at 1235) this datato one or more LB agents that need to process this data to specifyand/or update the load balancing rules that they maintain for their loadbalancers on their hosts. After 1235, the process determines (at 1240)whether it has received any more statistic and/or membership updates forwhich it needs to update its records. If not, the process remains at1240 until it receives statistics and/or membership updates, at whichtime it transitions back to 1210 to process the newly received statisticand/or membership updates.

In the embodiments described above by reference to FIGS. 10-12, the LBcontroller set 520 distributes global statistics to the LB agents, whichanalyze this data to specify and/or adjust the LB rules that theymaintain. In other embodiments, however, the LB controller set 520analyzes the global statistics that it gathers, and based on thisanalysis specifies and/or adjusts LB rules, which it then distributes tothe LB agents. In these embodiments, the LB agents simply store the LBrules or rule modifications that they receive from the LB controller setin the host-level LB rule storage 688 for distribution to the individualLB rule storages 440 of the load balancers 615.

FIG. 13 illustrates a process 1300 that shows the operation of the LBcontroller set for embodiments in which the LB controller set analyzesthe membership updates and/or global statistics, and in response to thisanalysis specifies and/or updates LB rules if needed. This process issimilar to the process 1200 of FIG. 12, except for the inclusion ofoperation 1312 and the replacement of operations 1225 and 1235 with theoperations 1325 and 1335.

At 1312, the process 1300 analyzes the membership and statistic recordsand if needed, specifies and/or updates the group memberships (e.g., theIP addresses) and/or the load balancing criteria (e.g., the weight ortime values) of one or more LB rules. This operation is similar to theoperation 1120 of the process 1100 of FIG. 11, except when performed bythe process 1300 of the LB controller set, the operation 1312 mightgenerate LB rules or rule updates for the load balancers of multiplehosts. From 1312, the process transitions to 1215, which was describedabove.

At 1325, the process 1300 determines whether it has to distribute thenewly specified and/or updated LB rules. If not, the process transitionsto 1230, which was described above. Otherwise, the process transitionsto 1335 to distribute the newly specified and/or updated LB rules to theLB agents of the hosts that have load balancers that need to enforce thespecified and/or updated LB rules. After 1335, the process transitionsto 1240, which was described above.

FIGS. 14-16 present several examples that illustrate how someembodiments dynamically adjust the spreading of traffic by adjusting theload balancing criteria and by adding/removing DCN VMs. Each of theseexamples is illustrated in terms of multiple operational stages thatshow several inline load balancers 1400 dynamically adjust how theyspread the data traffic from several webserver VMs 1405 to severalapplication server VMs 1410. In these examples, each load balancer 1400is associated with one webserver 1405, while the application server VMs1410 are part of one DCN group 1450 that is associated with one virtualaddress identifier. Also, the load balancers 1400, the web servers 1405,and the application server 1410 execute on one or more hosts. On thehosts, one or more LB agents 620 execute to exchange statistics with theLB controller set 520, in order to allow the load balancing operationsto be dynamically updated based on dynamically detected load conditions.For the sake of simplifying these figures, the LB agents 620 are notshown in FIGS. 14-16.

In three operational stages 1401-1403, FIG. 14 illustrates an examplewhere the load balancing criteria is adjusted based on dynamicallydetected load conditions. In this example, each load balancer 1400 usesa weighted round robin scheme to distribute the data messages from itsassociated webserver 1405. The weight values that control this schemeare adjusted by the LB agent(s) based on global load statistics that aresupplied by the LB controller set 520. These statistics specify the loadon the application server VMs 1410.

In the first operational stage 1401 of FIG. 14, each load balancer 1400evenly distributes the data messages of its webserver VMs 1405 among theapplication server VMs 1410. This even distribution is depicted in thisfigure by the designation of 10, 10, 10, 10, and 9 on the lines thatstart on the load balancer 1400 a and terminate on the applicationservers 1410. These numbers are the numbers of active data flows thatthe load balancer 1400 a is directing to the application servers 1410.As shown, the load balancer 1400 a in this stage bases its operation onthe weight values 1, 1, 1, 1, and 1. These weight values specify thatthe load balancer should evenly distribute to the five applicationservers 1410 the next five new data message flows from the webserver1405 a to the application server group 1450.

The first stage 1401 also shows the LB controller set 520 receivinglocal connection statistics from each of the load balancers 1400. Thesestatistics are gathered and relayed by the load balancers' LB agents,which are not shown in FIG. 14. The first stage 1401 also shows anexample of one of the provided local connection statistics, which is thelocal statistics 1420 that the load balancer 1400 a provides to the LBcontroller set 520. This local statistics 1420 show that the loadbalancer 1400 a currently has 10, 10, 10, 10, and 9 active flows that itis directing respectively to the application servers 1410 a-1410 e ofthe group 1450.

In different embodiments, the load balancers use different techniques toquantify the number of active flows that they are directing to eachapplication server 1410. In some embodiments, the load balancers timeout (i.e., remove) flows that are inactive (i.e., for which they havenot received any new data messages) after a particular duration of time.Other embodiments use other techniques to quantify the number of activeflows.

Instead of specifying the number of active flow to express the datatraffic load on the DCNs (i.e., the application servers in thisexample), other embodiments use other traffic metrics. For instance, theload balancers 1400 collect the number of data messages (e.g., datapackets) that they route to each application server 1410 in someembodiments. Other examples collect other traffic metrics such as TCPRTT and window size, retransmission, etc. Still other embodimentscollect other load metrics (such as round-trip delay, TCP window size,etc.) that express the load that each load balancer detects to each DCNto which the load balancer directs traffic. In some embodiments, the LBagents of the load balancers measure these other load metrics (e.g., theround-trip delay or TCP window size), while in other embodiments, theload balancers measure one or more of these load metrics (e.g., theround-trip delay or TCP window size).

The second stage 1402 shows the LB controller set 520 distributingglobal load statistics to the LB agents (not shown) of each of the loadbalancers 1400. The global load statistics in some embodiments is anaggregation of the local statistics that the load balancers provide(through the LB agent) to the LB controller set 520. The second stage1402 shows an example of the global connection statistics, which is theglobal statistics 1425 that the LB agent of the load balancer 1400 areceives from the LB controller set 520. As shown, the global statisticsin this example show the following numbers of active connections for thefive application servers 1410 a-1410 e: 131, 135, 101, 100, and 86.These numbers of connection represent the numbers of active flows thatall five load balancers 1400 are distributing to the five applicationservers 1410 a-1410 e from the five webservers 1405.

Like the gathered local statistics, the distributed global statisticsare different types of traffic and/or load metrics in other embodiments.In some embodiments, the distributed global statistics include for eachDCN in the DCN group, aggregated message traffic data that expresses thedata message traffic load on the DCN. Examples of such load data includethe number of data messages (e.g., number of packets) received by theDCN, number of flows processed by the DCN, number of data message bytesreceived by the DCN, etc. In some embodiments, the metrics can benormalized to units of time, e.g., per second, per minute, etc. Also, insome embodiments, the distributed global statistics express the datamessage load on each DCN in terms of a relative congestion percentagethat compares the load of the DCN to the load of other DCNs in thegroup.

In some embodiments, the distributed global statistics include anaggregated round trip delay (e.g., average round trip delay) to eachDCN, an aggregated TCP window size value (e.g., average TCP window size)for each DCN, etc. Also, in some embodiments, the distributed globalstatistics are partially or completely based on metrics that the LBcontroller set 520 gathers by interacting directly with the DCNs (e.g.with the application servers 1410). In some embodiments in which theglobal statistics are completely based on metrics directly gathered bythe LB controller set, the LB controller set does not gather statisticsthat the load balancers 1400 collect locally.

The second stage 1402 also shows the adjustment of the weight valuesthat the load balancer 1400 a uses to spread new flows to theapplication servers 1410. These weight values are adjusted by the LBagent(s) 620 based on the received global statistics 1425. The weightvalues after they are adjusted are 1, 1, 2, 2, 3. These weight valuesdirect the load balancer 1400 a to spread in a weighted round-robinapproach the next nine new data message flows as follows: 1 to the firstapplication server 1410 a, 1 to the second application server 1410 b, 2to the third application server 1410 c, 2 to the fourth applicationserver 1410 d, and 3 to the fifth application server 1410 e. Asmentioned above, some embodiments specify and use time period values inthe LB rules in order to allow the load balancers to gracefullytransition between different weight value sets to dynamically adjusttheir load balancing operations.

The third stage 1403 shows that after this adjustment of the weightvalues, the relative even distribution of flows by the load balancer1400 a becomes skewed towards the application servers 1410 that areassociated with the higher weight values, i.e., the application servers1410 c, 1410 d, and 1410 e. Specifically, this stage shows that once theweight values are adjusted, the number of flows (from the webservers1405 to the application servers 1410) goes from 20, 20, 20, 19, and 19,to 23, 23, 26, 26, and 28.

In the example illustrated in FIG. 14, the load balancing criteria(i.e., the weight values in this example) are adjusted by the LBagent(s) based on global statistics distributed by the LB controller set520. In other embodiments, however, the LB controller set adjusts anddistributes the load balancing criteria based on statistics that the LBcontroller set collects from the load balancers and/or from the DCNgroup(s). In these embodiments, the load balancers use the loadbalancing criteria distributed by the LB controller set to perform oradjust their load balancing operations. In some of these embodiments,the LB controller set also initially defines the LB rules with theinitial weight values, and distributes these rules to the load balancers(through the LB agents) for the load balancers to store and use.

In three operational stages 1501-1503, FIG. 15 illustrates an example ofadding a DCN to a DCN group to alleviate the traffic load on the DCNgroup members. This example follows the third stage 1403 of the exampleof FIG. 14. The first stage 1501 of FIG. 15 shows the addition (e.g.,the allotment or instantiation) of a sixth application sever 1410 f tothe application server group 1450. This sixth application server 1410 fhas been added to the group by the LB controller set 520 directing theVM managing controller set 525 to allot a previously created applicationserver VM to this group, or to instantiate a new application server VMfor this group.

The first stage 1501 also shows the LB controller set 520 providingglobal statistics 1520 and group update 1522 to the LB agents (notshown) of the load balancers 1400. The global statistics 1520 show thateach application server is currently handling about 50K flows, which inthis example is assumed to be near the threshold maximum number of flowsfor each application server. As shown, in this stage, the number offlows from load balancer 1400 a to the application servers is 20K, 18K,21K, 17K, and 19K.

The group update 1522 informs the load balancers that the sixthapplication server 1410 f has been added to the application server group1450. In response to this group update, the LB agent (not shown) of thewebserver 1405 a adjusts the weight values of the LB rule that loadbalancer 1400 a of this webserver enforces. As shown in the first stage1501, the adjusted weight values are 1, 1, 1, 1, 1, 1000. This weightvalue set directs the load balancer to assign the next 1005 new dataflows from the webserver 1405 a to the application servers 1410 a-1410 fbased on a weighted round robin scheme that assigns the next five newsflows to the applications servers 1410 a-1410 e, and then assign thenext 1000 flows to the application server 1410 f.

After receiving the group update 1522, the LB rules of the other loadbalancers of the other webservers 1405 are similarly adjusted by theirrespective LB agent(s). In response to these adjusted weight values, theload on the sixth application servers 1410 f starts to increase, whilethe load on the first five application servers 1410 a-1410 e starts todecrease, as shown in the second stage 1502. The second stage 1502 showsthe LB controller set providing updated global statistics 1525 to the LBagents (not shown) of the load balancers 1400. The updated globalstatistics 1525 shows that the load on the five application servers 1410a-1410 e has dropped to 40K, 39K, 41K, 38K and 39K, while the load onthe sixth application server 1410 f has risen to 18K. In this stage, thenumber of flows from load balancer 1400 a to the application servers isnow 14K, 12K, 13K, 15K, 16K, and 8K.

The second stage 1502 also shows that in response to the updated globalstatistics, the weight values for the load balancer 1400 a have beenadjusted to be 1, 1, 1, 1, 3. After receiving the global statistics1525, the weight values of the other load balancers of the otherwebservers 1405 are also adjusted by their respective LB agent(s). Thethird stage 1503 then shows that in response to these weight valueadjustments, the load across the application servers 1410 has reached44K, 42K, 43K, 45K, 46K, and 35K, as indicated in the updated globalstatistics 1535. In this stage, the number of flows from load balancer1400 a to the application servers is now 12K, 12K, 13K, 14K, 13K, and13K.

In three operational stages 1601-1603, FIG. 16 illustrates an example ofremoving a DCN from a DCN group when fewer DCNs are needed to handle theload on the DCN group. This example follows the third stage 1503 of theexample of FIG. 15. The first stage 1601 of FIG. 16 shows the LBcontroller set 520 providing global statistics 1620 and group update1622 to the LB agents (not shown) of the load balancers 1400. The globalstatistics 1620 show that the application servers 1410 respectivelyhandling 22K, 26K, 27K, 28K, 28K, and 26K flows. As shown, in thisstage, the number of flows from load balancer 1400 a to the applicationservers is 6K, 7K, 10K, 9K, 10K and 10K.

The first stage 1601 also shows the LB controller set 520 providing agroup update 1622 that informs the load balancers that the firstapplication server 1410 a should be removed from the application servergroup 1450. In response to this group update, the LB agent (not shown)of the webserver 1405 a adjusts the weight values of the LB rule thatload balancer 1400 a of this webserver enforces. As shown in the firststage 1601, the adjusted weight values are 0, 2, 1, 1, 1, 1. This weightvalue set directs the load balancer to assign the next 6 new data flowsfrom the webserver 1405 a to the application servers 1410 b-1410 f basedon a weighted round robin scheme that assigns the next two news flows tothe applications server 1410 b, and then assign the next four flowsindividually to each of the four application servers 1410 c-1410 f.

After receiving the group update 1622, the LB rules of the other loadbalancers of the other webservers 1405 are similarly adjusted by theirrespective LB agent(s). In response to these adjusted weight values, theload on the first application server 1410 a starts to decrease, whilethe load on the other five application servers 1410 b-1410 f starts toincrease, as shown in the second stage 1602. The second stage 1602 showsthe LB controller set providing updated global statistics 1625 to the LBagents (not shown) of the load balancers 1400. The updated globalstatistics 1625 shows that the load on the application server 1410 a hasdropped down to 12K flows, while the load on the application servers1410 b-1410 f has increased to 30K, 32K, 31K, 32K and 30K flows. In thisexample, the load on the application server 1410 a does not immediatelyfall to zero because this server continues to receive data messages forflows that it has been processing.

The second stage also shows the number of flows from load balancer 1400a to the application servers to now be 5K, 8K, 9K, 8K, 10K, and 9K. Thesecond stage 1602 further shows that in response to the updated globalstatistics, the weight values for the load balancer 1400 a have beenadjusted to be 0, 1, 1, 1, 1, 1. After receiving the global statistics1625, the weight values of the other load balancers of the otherwebservers 1405 are also adjusted by their respective LB agent(s).

The third stage 1603 then shows that in response to these weight valueadjustments, the application server 1410 a has effectively been removedfrom the DCN group 1450 as it no longer receives any flows from the loadbalancers 1400. This stage also shows that the load on the otherapplication servers 1410 b-f has reached 40K, 39K, 41K, 38K and 39Kflows, as indicated in the updated global statistics 1635. In thisstage, the number of flows from load balancer 1400 a to the applicationservers is now 0, 12K, 13K, 14K, 13K, and 13K.

Examples above show the addition of new DCNs to alleviate the trafficload. In some embodiments, the load on the DCNs can be adjusted byadding or removing SCN VMs. Also, even though the LB rules in theabove-described examples include weight values that facilitate the loadbalancers dynamic adjustment of the load, one of ordinary skill in theart will realize that in other embodiments the load balancers use othermechanisms for dynamically adjusting the data traffic load based ondynamically detected load conditions.

In the above-described examples, the load balancers are described asbalancing the data traffic between different layers of data compute endnodes (DCENs), such as webservers, application servers and databaseservers. However, in some embodiments, the distributed load balancingarchitecture can be used to load balance the data traffic to and frommiddlebox service nodes. In other words, the DCNs in the DCN group insome embodiments can be middlebox service nodes (such as firewalls,intrusion detectors, WAN optimizers, etc.).

Also, as illustrated in FIG. 17, the inline load balancers in someembodiments can be configured to route data messages that are sent toDCENs initially to a set of middlebox service nodes. In this example,inline load balancers 1700 (associated with the webserver VMs 1705)direct the data traffic that web servers 1705 send to applicationservers 1710, to firewall middlebox VMs 1720. In directing the datamessages to the firewalls 1720, the inline load balancers perform loadbalancing operations that spread the data message load among thefirewalls 1720. Once processed by the firewalls 1720, thefirewall-filtered data messages are distributed by the inline loadbalancers 1730 (associated with the firewall VMs 1720) to theapplication servers 1710. As shown, in this example, the firewallservers are service VMs executing on the same hosts as the webserversand application servers.

To direct to the firewall VMs the data traffic that is addressed to theapplication servers' virtual address (e.g., VIP), the load balancers1700 in some embodiments (1) perform a virtual address (e.g., a VIP)translation that replaces the application server virtual address withthe firewall VM's virtual address, and then (2) spread the received datatraffic amongst the firewall VMs based on their load balancing criteria.In some embodiments, the load balancers 1700 address translation insertsidentifiers in the message identifiers (e.g., in the packet header) thatallows the firewall VMs 1720 and load balancers 1730 to determine thattheir received messages are directed to application servers 1710. Tomake this determination, the load balancers 1730 are configured withrules that enable the load balancers to associate the received datamessages with the application servers 1710 in some embodiments.

As mentioned above, an inline load balancer 615 of a VM 605 can performmultiple different load balancing operations for multiple differentgroups of DCNs. This is because the load balancer 615 can apply the loadbalancing rules of multiple different groups of DCNs. These rules arestored in the load balancing data storage 440, as described above byreference to FIGS. 6-8.

FIG. 18 illustrates that one inline load balancer can form multipledifferent distributed load balancers with multiple different sets ofinline load balancers. FIG. 18 presents two sets of inline loadbalancers that distribute the data messages of two different sets 1805and 1810 of VMs to two different groups of DCNs 1820 and 1825. One VM,VM1, is part of both sets 1805 and 1810 of VMs. Each inline loadbalancer is analogous to the inline load balancer 615 of FIG. 6.

As shown in FIG. 18, the inline load balancer 1815 of VM1 enforces loadbalancing rules 1850 and 1855 that are stored in its load balancingstorage 1840. These load balancing rule 1850 and 1855 direct the loadbalancer 1815 to distribute data messages of VM1 that are directedrespectively to DCN groups 1820 and 1825 to the DCNs in these groups.Also, in this example, the inline load balancer 1870 of the virtualmachine VM2 enforces a LB rule for distributing data messages for DCNgroup 1820, while the inline load balancer 1875 of the virtual machineVM3 enforces a LB rule for distributing data messages for DCN group1825. The LB rules of the inline load balancers 1815 and 1870 of VM1 andVM2 for DCN group 1820 can have identical LB criteria or different LBcriteria. Similarly, the LB rules of the inline load balancers 1815 and1870 of VM1 and VM3 for DCN group 1825 can have identical LB criteria ordifferent LB criteria. These load balancing rules (e.g., rules 1850 and1855) and their associated load balancing storage (e.g., storage 1840)are analogous to the load balancing rules 700 and 800 and the loadbalancing storage 440 of FIGS. 6-8.

As shown in FIG. 18, the inline load balancers (e.g., 1815 and 1870 ofVM1 and VM2) of VM group 1805 form a distributed load balancer 1880 thatdistributes the data messages from VMs of group 1805 amongst the DCNs ofDCN group 1820. Similarly, as shown, the inline load balancers (e.g.,1815 and 1875 of VM1 and VM3) of VM group 1810 form a distributed loadbalancer 1890 that distributes the data messages from VMs of group 1810amongst the DCNs of DCN group 1825.

Each distributed load balancer 1880 or 1890 is logical construct as itis not one item in the physical world, but rather conceptuallyrepresents one set of load balancing operations that a group ofassociated inline load balancers performs to distribute the data messageload on a DCN group. In this distributed approach, each inline loadbalancer only needs to store the load balancing rules of the distributedload balancer that it implements. In other words, each inline loadbalancer in this distributed approach only needs to store the loadbalancing rules of the DCN-group data messages that its associated VMmight send out. Also, in this distributed approach, each inline loadbalancer needs to only maintain in its connection data store (e.g.,connection data storage 690) the flow connection states of the datamessage flows sent by the load balancer's associated VM. Because of allof these reasons, the inline load balancers of some embodiments are fastand efficient as they maintain small LB rule and connection state datastorages that they can search quickly.

In the example illustrated in FIG. 18, the inline load balancer 1815 isshown to be part of two distributed load balancers 1880 and 1890 bybeing part of two sets of associated load balancers, one for the VMgroup 1805 and another for the VM group 1810. In other examples, aninline load balancer can be part of any arbitrary number N ofdistributed load balancers, when with another set of N inline loadbalancers it enforces N load balancing rules for data messages that arebeing directed to N different DCN groups.

FIG. 19 illustrates another example that illustrates that the inlineload balancers of some embodiments can differently translate the virtualaddresses of data messages to different groups of DCNs. Specifically,this figure illustrates five inline load balancers 1900 of fivewebservers 1905 that direct and load balance data messages to a firstVIP associated with a first group 1910 of application servers to theapplication servers 1915 and 1920 of this group 1910, while directingand load balancing data messages to a second VIP associated with asecond group 1930 of application servers to the application servers 1935and 1940 of this group 1930.

In some embodiments, the inline load balancers differently direct andload balance data messages that are addressed to the same virtualaddress. For instance, some embodiments define priority sub-groupswithin an addressed DCN group, and load balance different priority datamessages to different sub-groups based on their priority. For example,FIG. 20 illustrates a set of inline load balancers 2000 that directwebserver data messages to a group 2050 of application servers to eithera high-priority sub-group 2040 of application servers or a low prioritygroup 2045 of application servers based on the assessed priority of thedata messages.

In different embodiments, the load balancers 2000 assess the priority ofthe data messages from the webservers 2005 differently. For instance, insome embodiments, the load balancers assess the priority of the datamessages based on identity of the sources from which the webserverreceived the data messages. After assessing the priority of the datamessages, the load balancers direct the received data messages to theapplication server sub-group with the corresponding priority.

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 21 conceptually illustrates a computer system 2100 with which someembodiments of the invention are implemented. The computer system 2100can be used to implement any of the above-described hosts, controllers,and managers. As such, it can be used to execute any of the abovedescribed processes. This computer system includes various types ofnon-transitory machine readable media and interfaces for various othertypes of machine readable media. Computer system 2100 includes a bus2105, processing unit(s) 2110, a system memory 2125, a read-only memory2130, a permanent storage device 2135, input devices 2140, and outputdevices 2145.

The bus 2105 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of thecomputer system 2100. For instance, the bus 2105 communicativelyconnects the processing unit(s) 2110 with the read-only memory 2130, thesystem memory 2125, and the permanent storage device 2135.

From these various memory units, the processing unit(s) 2110 retrieveinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments. Theread-only-memory (ROM) 2130 stores static data and instructions that areneeded by the processing unit(s) 2110 and other modules of the computersystem. The permanent storage device 2135, on the other hand, is aread-and-write memory device. This device is a non-volatile memory unitthat stores instructions and data even when the computer system 2100 isoff. Some embodiments of the invention use a mass-storage device (suchas a magnetic or optical disk and its corresponding disk drive) as thepermanent storage device 2135.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, etc.) as the permanent storage device. Like the permanentstorage device 2135, the system memory 2125 is a read-and-write memorydevice. However, unlike storage device 2135, the system memory is avolatile read-and-write memory, such a random access memory. The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the invention's processes are stored inthe system memory 2125, the permanent storage device 2135, and/or theread-only memory 2130. From these various memory units, the processingunit(s) 2110 retrieve instructions to execute and data to process inorder to execute the processes of some embodiments.

The bus 2105 also connects to the input and output devices 2140 and2145. The input devices enable the user to communicate information andselect commands to the computer system. The input devices 2140 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 2145 display images generated by thecomputer system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 21, bus 2105 also couples computer system 2100to a network 2165 through a network adapter (not shown). In this manner,the computer can be a part of a network of computers (such as a localarea network (“LAN”), a wide area network (“WAN”), or an Intranet, or anetwork of networks, such as the Internet. Any or all components ofcomputer system 2100 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”,“processor”, and “memory” all refer to electronic or other technologicaldevices. These terms exclude people or groups of people. For thepurposes of the specification, the terms display or displaying meansdisplaying on an electronic device. As used in this specification, theterms “computer readable medium,” “computer readable media,” and“machine readable medium” are entirely restricted to tangible, physicalobjects that store information in a form that is readable by a computer.These terms exclude any wireless signals, wired download signals, andany other ephemeral or transitory signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. For instance, while the load balancingprocesses were described above by reference to several hostarchitecture, one of ordinary skill in the art will realize that theseprocesses could be implemented in a variety of different architecturesthat load balance messages at variety of different locations along theiregress path out of the host. For instance, in some embodiments, the loadbalancing processes are implemented in the PNIC of the host. In otherwords, the PNIC of the host in some embodiments examines the VM messagesto determine whether it should load balance them before sending them outof the host or sending them to their destination GVMs.

In many of the above-described examples, the virtual addresses are VIPs,which the load balancers replace by physical IP addresses of the DCNVMs. However, one of ordinary skill in the art will realize that, inother embodiments, the virtual addresses are different types ofaddresses and the load balancers perform other address translationoperations. For example, in some embodiments, the load balancertranslates a virtual port address to a physical port address (i.e.,performs L4 address translation operations), instead of or inconjunction with performing the IP network address translation (toreplace the VIP with a physical IP address). In still other embodiments,the load balancer directs a data message to a DCN in a DCN group throughMAC redirection operation, which replaced one MAC address with the MACaddress of the DCN that should received the data messages. In someembodiments, the DCNs are connected to one distributed logical switchthat logically spans multiple hosts, and the MAC redirection directs adata message that is addressed to one port of the logical switch toanother port of the logical switch.

In many of the above-described examples, an LB agent adjusts the loadbalancing criteria for the load balancers that execute on its host basedon the data distributed by the controller set. One of ordinary skillwill realize that in other embodiments, the load balancers themselvesadjust their load balancing criteria based on the data distributed bythe controller set.

This specification refers throughout to computational and networkenvironments that include virtual machines (VMs). However, virtualmachines are merely one example of a compute node, also referred to asaddressable nodes. Some embodiments of the invention are equallyapplicable to any computing node that utilizes a port abstractiondefined on a host computing device to allow multiple programs thatexecute on the host to share common resources on the host. As such, thecompute nodes in some embodiments may include non-virtualized physicalhosts, virtual machines, containers that run on top of a host operatingsystem without the need for a hypervisor or separate operating system,and hypervisor kernel network interface modules.

VMs, in some embodiments, operate with their own guest operating systemson a host using resources of the host virtualized by virtualizationsoftware (e.g., a hypervisor, virtual machine monitor, etc.). The tenant(i.e., the owner of the VM) can choose which applications to operate ontop of the guest operating system. Some containers, on the other hand,are constructs that run on top of a host operating system without theneed for a hypervisor or separate guest operating system. In someembodiments, the host operating system uses name spaces to isolate thecontainers from each other and therefore provides operating-system levelsegregation of the different groups of applications that operate withindifferent containers. This segregation is akin to the VM segregationthat is offered in hypervisor-virtualized environments that virtualizesystem hardware, and thus can be viewed as a form of virtualization thatisolates different groups of applications that operate in differentcontainers. Such containers are more lightweight than VMs.

Hypervisor kernel network interface modules, in some embodiments, is anon-VM DCN that includes a network stack with a hypervisor kernelnetwork interface and receive/transmit threads. One example of ahypervisor kernel network interface module is the vmknic module that ispart of the ESXi™ hypervisor of VMware, Inc.

One of ordinary skill in the art will recognize that while thespecification refers to VMs, the examples given could be any type ofDCNs, including physical hosts, VMs, non-VM containers, and hypervisorkernel network interface modules. In fact, the example networks couldinclude combinations of different types of DCNs in some embodiments.

A number of the figures (e.g., FIGS. 9-13) conceptually illustrateprocesses. The specific operations of these processes may not beperformed in the exact order shown and described. The specificoperations may not be performed in one continuous series of operations,and different specific operations may be performed in differentembodiments. Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process. In view of theforegoing, one of ordinary skill in the art would understand that theinvention is not to be limited by the foregoing illustrative details,but rather is to be defined by the appended claims.

We claim:
 1. A non-transitory machine readable medium storing a programfor managing a set of service destination compute nodes (DCNs) thatprocess data messages from a set of source compute nodes (SCNs), theprogram comprising sets of instructions for: analyzing message trafficload data relating to data messages that are distributed to the DCNs ofthe DCN sets; based on the analyzed message traffic load data, directinga set of one or more controllers to modify the DCN set's membership. 2.The non-transitory machine readable medium of claim 1, wherein the setof instructions for directing the controller set comprises a set ofinstructions for directing the controller set to remove a DCN from theDCN set when the analyzed message traffic load data indicates that oneor more DCNs in the DCN set are being underutilized.
 3. Thenon-transitory machine readable medium of claim 1, wherein the set ofinstructions for directing the controller set comprises a set ofinstructions for directing the controller set to add a new DCN to theDCN set when the data message load on the DCN set or on one DCN in theDCN set exceeds a threshold value.
 4. The non-transitory machinereadable medium of claim 3, wherein the controller set adds a new DCN byprovisioning a new DCN and adding the new DCN to the DCN set.
 5. Thenon-transitory machine readable medium of claim 3, wherein thecontroller set adds a new DCN by allocating a previously provisioned DCNto the DCN set.
 6. The non-transitory machine readable medium of claim1, wherein the program further comprises a set of instructions forcollecting message traffic load data relating to data messages that aredistributed to the DCNs of the DCN sets.
 7. The non-transitory machinereadable medium of claim 6, wherein the set of instructions forcollecting message traffic load data comprises a set of instructions forcollecting message traffic load data from load balancers that aredeployed in the egress datapaths of the SCNs and that distribute thedata messages from the SCNs to the DCNs in the DCN set.
 8. Thenon-transitory machine readable medium of claim 7, wherein at least oneload balancer for at least one group of SCNs (i) executes on one hostcomputing device with the group of SCNs, and (ii) intercepts datamessages sent from the SCNs in the SCN group and directs in a loadbalanced manner the intercepted data messages to the DCNs of the DCN setwhen data messages need to be processed by the DCN set.
 9. Thenon-transitory machine readable medium of claim 1, wherein the SCN groupincludes only one SCN.
 10. The non-transitory machine readable medium ofclaim 1, wherein the SCN group includes more than one SCN.
 11. Thenon-transitory machine readable medium of claim 7, wherein a pluralityof SCNs and a plurality of load balancers execute on a set of hostcomputing devices; wherein each load balancer in the plurality of loadbalancers is associated with one SCN, executes on the same hostcomputing device at its associated SCN, intercepts data messages sentfrom its associated SCN and directs in a load balanced manner theintercepted data messages to the DCNs of the DCN set when data messagesneed to be processed by the DCN set.
 12. The non-transitory machinereadable medium of claim 1, wherein the set of instructions fordirecting the controller comprises automatically directing thecontroller to modify DCN set membership without receiving any userinput.
 13. The non-transitory machine readable medium of claim 1,wherein the DCNs in the DCN set perform a middlebox service on the datamessages that they process.
 14. A method for managing a set of servicedestination compute nodes (DCNs) that process data messages from a setof source compute nodes (SCNs), the method comprising: analyzing messagetraffic load data relating to data messages that are distributed to theDCNs of the DCN sets; based on the analyzed message traffic load data,directing a controller to modify membership of DCNs in the DCN set. 15.The method of claim 14, wherein directing the controller comprisesdirecting the controller to remove a DCN from the DCN set when theanalyzed message traffic load data indicates that one or more DCNs inthe DCN set are being underutilized.
 16. The method of claim 14, whereindirecting the controller comprises directing the controller to add a newDCN to the DCN set when analyzed message traffic load data indicatesthat one or more DCNs in the DCN set are being overutilized.
 17. Themethod of claim 16, wherein the DCNs comprise virtual machines (VMs) thecontroller is a virtualization manager for managing a plurality of VMsthat execute on a set of host computing devices; the virtualizationmanager adds a new DCN by instantiating a new VM and adding the new VMto the DCN set.
 18. The method of claim 16, wherein the DCNs comprisevirtual machines (VMs) the controller is a virtualization manager formanaging a plurality of VMs that execute on a set of host computingdevices; the virtualization manager adds a new DCN by allocating apreviously instantiated VM to the DCN set.
 19. The method of claim 14further comprising collecting message traffic load data from loadbalancers that are deployed in the egress datapaths of the SCNs and thatdistribute the data messages from the SCNs to the DCNs in the DCN set.20. The method of claim 19, wherein at least one load balancer for atleast one group of SCNs (i) executes on one host computing device withthe group of SCNs, and (ii) intercepts data messages sent from the SCNsin the SCN group and directs in a load balanced manner the intercepteddata messages to the DCNs of the DCN set when data messages need to beprocessed by the DCN set.
 21. The non-transitory machine readable mediumof claim 19, wherein a plurality of SCNs and a plurality of loadbalancers execute on a set of host computing devices; wherein each loadbalancer in the plurality of load balancers is associated with one SCN,executes on the same host computing device at its associated SCN,intercepts data messages sent from its associated SCN and directs in aload balanced manner the intercepted data messages to the DCNs of theDCN set when data messages need to be processed by the DCN set.
 22. Thenon-transitory machine readable medium of claim 14, wherein the set ofinstructions for directing the controller comprises automaticallydirecting the controller to modify DCN set membership without receivingany user input.